-
Notifications
You must be signed in to change notification settings - Fork 804
Make DXC and DXV use new DxcDllExtValidationLoader
to load external validator before validation.
#7749
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Make DXC and DXV use new DxcDllExtValidationLoader
to load external validator before validation.
#7749
Conversation
✅ With the latest revision this PR passed the C/C++ code formatter. |
DxcDllExtValidationLoader
to load external validator before validation.DxcDllExtValidationLoader
to load external validator before validation.
6c69325
to
78cace7
Compare
lib/DxcSupport/dxcapi.extval.cpp
Outdated
UINT32 newArgCount = ArgCount + 1; | ||
std::vector<LPCWSTR> newArgs; | ||
newArgs.reserve(newArgCount); | ||
|
||
// Copy existing args | ||
for (unsigned int i = 0; i < ArgCount; ++i) { | ||
newArgs.push_back(pArguments[i]); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I think rewriting this as below is equally performant and certainly cleaner
UINT32 newArgCount = ArgCount + 1; | |
std::vector<LPCWSTR> newArgs; | |
newArgs.reserve(newArgCount); | |
// Copy existing args | |
for (unsigned int i = 0; i < ArgCount; ++i) { | |
newArgs.push_back(pArguments[i]); | |
} | |
std::vector<LPCWSTR> newArgs = {pArguments, pArguments + ArgCount}; |
lib/DxcSupport/dxcapi.extval.cpp
Outdated
if (pResult == nullptr) | ||
return E_POINTER; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit:
if (pResult == nullptr) | |
return E_POINTER; | |
if (!pResult) | |
return E_POINTER; |
lib/DxcSupport/dxcapi.extval.cpp
Outdated
hr = evh->QueryInterface(riid, reinterpret_cast<void **>(pResult)); | ||
return hr; | ||
|
||
} else if (clsid == CLSID_DxcValidator) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: you can delete the else
here. Might even be worth moving the entire if
to the top of this block since its quite short comparted to the other one
lib/DxcSupport/dxcapi.extval.cpp
Outdated
IUnknown **pResult) { | ||
if (DxilExtValSupport.IsEnabled() && clsid == CLSID_DxcValidator) | ||
return DxilExtValSupport.CreateInstance2(pMalloc, clsid, riid, pResult); | ||
if (pResult == nullptr) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit:
if (pResult == nullptr) | |
if (!pResult) |
tools/clang/tools/dxclib/dxc.cpp
Outdated
if (DXC_FAILED(ValHR)) { | ||
if (SUCCEEDED(pCompileResult->QueryInterface(&pResult))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: it looks like these can be a single if statement
lib/DxcSupport/dxcapi.extval.cpp
Outdated
// validation scenario, so there is no point in also running | ||
// the internal validator. This wrapper wraps both the | ||
// IDxcCompiler and IDxcCompiler3 interfaces. | ||
class ExternalValidationHelper : public IDxcCompiler, public IDxcCompiler3 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For one, you'll need to wrap all the interfaces that the compiler object implements so you don't lose the object wrapping for any of them.
But also, IDxcCompiler
and IDxcCompiler3
should be incompatible to derive from and implement in the same class due to the nature of COM not supporting separate overloaded methods of the same name. This requires a separate object to implement the other interfaces.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So to be clear, are you suggesting one parent class, ExternalValidationHelper, that implements 4 interfaces and has 4 wrapper member objects, that each implement one interface: IDxcCompiler, IDxcCompiler2, IDxcCompiler3, and IDxcValidator? Your comment above, "This code should all be inside the compiler object wrapper.", makes me think that this class should also implement the IDxcValidator interface, is that correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But also,
IDxcCompiler
andIDxcCompiler3
should be incompatible to derive from and implement in the same class due to the nature of COM not supporting separate overloaded methods of the same name.
It's lucky that IDxcCompiler and IDxcCompiler3 don't actually have any identical methods in them so that this isn't a problem.
IDxcCompiler::Compile takes 10 args, while IDxcCompiler3::Compile takes 7
Preprocess only exists on IDxcCompiler
IDxcCompiler::Disassemble takes 2 args while IDxcCompiler3::Disassemble take 3.
If you only try and implement one of these when you derive from both interfaces you'll get a warning about hiding the base class version.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting. I thought it was not allowed, hence the hoops we jumped through multiple times for multiple implementations of the compiler object supporting both interfaces to avoid deriving from both on the same implementation class.
In any case the first point stands, that there are more interfaces to support than these two.
You'll want to derive from each of these:
IDxcCompiler2
IDxcCompiler3
IDxcLangExtensions3
IDxcContainerEvent
IDxcVersionInfo3
IDxcVersionInfo2
(orIDxcVersionInfo
ifSUPPORT_QUERY_GIT_COMMIT_INFO
not defined)
And no need to derive from IDxcCompiler
when deriving from IDxcCompiler2
, since IDxcCompiler2
already derives from IDxcCompiler
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The implementation in this PR will cause failures if any of the tests attempt to use these interfaces. I don't think we need to wrap additional ones for the sake of it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Like Damyan said, I think I'll worry about deriving from each of those in a future PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are quite a few issues, and as I tried to write something that could express the exact logic necessary for the arguments, I started rewriting the ExternalValidationHelper implementation to also fix many of the other issues. I put a branch up, so here's the link to the commit:
See the commit for the general list of issues this rewrite fixes.
c6ac790
to
eb69e26
Compare
fc32e76
to
1d8e2ef
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this still looks good to me. It's very hard for me to review, because the rebase means that I can't easily tell what changed since my last review. Since we squash all our commits into main there's no real benefit to rebasing rather than merging with main.
lib/DxcSupport/dxcapi.extval.cpp
Outdated
if (Result) | ||
return Result->QueryInterface(Riid, ValResult); | ||
else | ||
return E_FAIL; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit:
if (Result) | |
return Result->QueryInterface(Riid, ValResult); | |
else | |
return E_FAIL; | |
if (!Result) | |
return E_FAIL; | |
return Result->QueryInterface(Riid, ValResult); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Submitting a block of comments while I continue to review.
lib/DxcSupport/dxcapi.extval.cpp
Outdated
if (Result) | ||
return Result->QueryInterface(Riid, ValResult); | ||
else | ||
return E_FAIL; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suggest returning E_POINTER as I think that's what this HRESULT is intended to reflect?
That is, we expect that the caller gives us a valid IDxcOperationResult*.
lib/DxcSupport/dxcapi.extval.cpp
Outdated
|
||
// No validation needed; just set the result and return. | ||
if (!NeedsValidation) | ||
return UseResult(CompileResult); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The UseResult name seems misleading to me? It really looks like it's just a glorified QueryInterface call?
I think it would be cleaner to
- Remove the UseResult lambda
- Add a check to return E_INVALID_ARG if CompileResult is an invalid pointer at the start of doValidation.
- Update line 59 to 'return CompileResult->QueryInterface(Riid, ValResuilt);
- Same update to line 67.
- Same update for line 83.
- Same update on line 87.
lib/DxcSupport/dxcapi.extval.cpp
Outdated
if (ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor, | ||
&ValidatorVersionMinor)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if (ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor, | |
&ValidatorVersionMinor)) { | |
if (FAILED(ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor, | |
&ValidatorVersionMinor))) { |
lib/DxcSupport/dxcapi.extval.cpp
Outdated
#endif | ||
!Opts.DumpDependencies && !Opts.VerifyDiagnostics && | ||
Opts.Preprocess.empty(); | ||
bool ProduceFullContainer = ProduceDxModule && !Opts.CodeGenHighLevel; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bool ProduceFullContainer = ProduceDxModule && !Opts.CodeGenHighLevel; | |
const bool ProduceFullContainer = ProduceDxModule && !Opts.CodeGenHighLevel; |
return DoBasicQueryInterface<IDxcCompiler, IDxcCompiler2, IDxcCompiler3>( | ||
NewWrapper.p, Iid, ResultObject); | ||
} catch (...) { | ||
return E_FAIL; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the exception expected here would be due to an allocation failure.
So would ERROR_OUTOFMEMORY be a more appropriate HRESULT?
if (ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor, | ||
&ValidatorVersionMinor)) { | ||
DXASSERT(false, "Failed to get validator version"); | ||
return E_FAIL; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I like to try and avoid E_FAIL because its pretty generic.
There is an 'ERROR_VERSION_PARSE_ERROR' you could use here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gonna stick to E_FAIL since there is no definition for that Error code in non-windows.
Same applies to ERROR_OUTOFMEMORY
lib/DxcSupport/dxcapi.extval.cpp
Outdated
// The ExtValidationArgHelper class helps manage arguments for external | ||
// validation, as well as perform the validation step if needed. | ||
class ExtValidationArgHelper { | ||
std::vector<std::wstring> ArgStorage; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we generally do all publics first and then all privates second?
Should we move the member variables to the private section?
lib/DxcSupport/dxcapi.extval.cpp
Outdated
// IDxcValidator interface to validate a successful compilation result, when | ||
// validation would normally be performed. | ||
class ExternalValidationCompiler : public IDxcCompiler2, public IDxcCompiler3 { | ||
private: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: public first for consistency with the rest of this file
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually the reason this broke that consistency is because there's a special rule in linux / macos builds on initialization order, and the Compiler object can't be initialized after the m_pAlloc field, which was originally taken care of in the previous private block. So I will leave this as is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there's a special rule in linux / macos builds on initialization order
There's not a special rule, it's just that MSVC doesn't warn if you mismatch the order that things appear in the constructor's initialization list with the order that they are actually initialized (ie the order they appear in the class itself).
If you wanted to put the private block at the end then you'd just need to update the constructor to list m_pMalloc
first (since it'd be listed first in the class as the field is added by the DXC_MICROCOM_TM_REF_FIELDS
macro).
Or you could have your private block at the end look like this and not need to modify the constructor:
private:
CComPtr<IDxcValidator> Validator;
IID CompilerIID;
CComPtr<IUnknown> Compiler;
DXC_MICROCOM_TM_REF_FIELDS()
lib/DxcSupport/dxcapi.extval.cpp
Outdated
CComPtr<IDxcBlob> CompiledBlob; | ||
IFR(CompileResult->GetResult(&CompiledBlob)); | ||
|
||
// If no compiled blob; just return the compile result. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: comment doesn't seem accurate. We're returning the HRESULT of QI. That doesn't seem like a compile result?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This language is a bit overloaded, I meant return through an out-parameter, by placing the result into ValResult.
I can rewrite this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The main thing that threw me off is that you wouldn't generally expect a QI call to do anything other than get you a pointer to the requested interface. The fact that the private implementation of QI is doing more than that is a little weird.
// Return the validation result if it failed. | ||
HRESULT HR; | ||
IFR(TempValidationResult->GetStatus(&HR)); | ||
if (FAILED(HR)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see uses of DXC_FAILED and FAILED. Hadn't seen DXC_FAILED before but it looks like they do the same thing.
We should just use one.
!Opts.VerifyDiagnostics && | ||
Opts.Preprocess.empty(); | ||
const bool ProduceFullContainer = ProduceDxModule && !Opts.CodeGenHighLevel; | ||
NeedsValidation = ProduceFullContainer && !Opts.DisableValidation; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I found it a little confusing to follow the logic for toggling NeedsValidation at first. Wondering if it might be a little cleaner and more readable if you add a helper function that toggles it for you.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I want to leave it as is. There are intermediate variables that are used to determine whether or not to modify the new args, and that dependency would reach beyond the scope of the new helper function if I make a new one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ProduceDxModule
could be turned into a const accessor method on DxcOpts
, then dxcompilerobj.cpp
could use that instead of keeping these duplicated blocks in sync across projects. The ProduceFullContainer
is simpler based on that one and could be separate, or it could also be const accessors that use the ProduceDxModule
one.
NeedsValidation
could also have an accessor, and could encode the root signature logic below, using the new ShaderModel::ParseTargetProfile
method merged from your recently merged PR. DxcOpts
already has a IsRootSignatureProfile()
for initializing the RootSignatureOnly
flag, though I don't think that flag is used here, so it could be removed instead.
I agree with @alsepkow, it would be cleaner, plus locating it in DxcOpts
makes more sense.
// correct for the method call. | ||
// This will either be casting to the original interface retrieved by | ||
// QueryInterface, or to one from which that interface derives. | ||
template <typename T> T *castSafe() const { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see how this, as it is, adds any value.
If we want safe casting, we could use a template method specialization approach for base interfaces to try casting to derived interfaces first.
Here's a safe casting pattern using the castSafe method:
// A recursive cast using specializations to handle derived interfaces.
template <typename T> T *cast() const { return castSafe<T>(); }
// Specialize cast to recurse into derived interfaces.
template <> IDxcCompiler *cast<IDxcCompiler>() const {
if (IDxcCompiler2 *Result = cast<IDxcCompiler2>())
return Result;
return castSafe<IDxcCompiler>();
}
Now you can use cast<Interface>()
instead of castUnsafe<Interface>()
for all cases.
@@ -0,0 +1,22 @@ | |||
// REQUIRES: v1.6.2112 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This string seems too generic for identifying an external validator version. Shouldn't it be more like "dxilval-v1.6.2112" or something like that?
// CHECK: error: RWStructuredBuffers may increment or decrement their counters, but not both. | ||
// CHECK: note: at '%3 = call i32 @dx.op.bufferUpdateCounter(i32 70, %dx.types.Handle %1, i8 1)' in block '#0' of function 'main'. | ||
// CHECK: Validation failed. | ||
|
||
RWStructuredBuffer<uint> UAV : register(u0); | ||
|
||
[numthreads(64, 1, 1)] | ||
void main(uint tid : SV_GroupIndex) { | ||
UAV[UAV.IncrementCounter()] = tid; | ||
UAV[UAV.DecrementCounter()] = tid * 2; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought we discussed the approach before and you were going to use a different approach, where you have a simple valid shader, with the explicit validator version that's too high by 1. That would match the comments, but this shader still wouldn't pass a newer validator due to the increment and decrement, so it doesn't behave the way the comments above describe.
It would be simpler and clearer if you changed to this, I think:
// CHECK: error: RWStructuredBuffers may increment or decrement their counters, but not both. | |
// CHECK: note: at '%3 = call i32 @dx.op.bufferUpdateCounter(i32 70, %dx.types.Handle %1, i8 1)' in block '#0' of function 'main'. | |
// CHECK: Validation failed. | |
RWStructuredBuffer<uint> UAV : register(u0); | |
[numthreads(64, 1, 1)] | |
void main(uint tid : SV_GroupIndex) { | |
UAV[UAV.IncrementCounter()] = tid; | |
UAV[UAV.DecrementCounter()] = tid * 2; | |
} | |
[numthreads(1, 1, 1)] | |
void main() {} |
Then it would be clear that the point of the test is to see the matching validator version error, which proves we are using the expected validator.
config.environment.update({'DXC_DXIL_DLL_PATH' : dxc_dll_path}) | ||
config.available_features.add("dxc_dxil_dll_path") | ||
dxc_version = lit_config.params.get('DXC_VERSION', None) | ||
config.available_features.add(dxc_version) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What exactly is DXC_VERSION
? Why would we add the version value without a clear prefix identifying what it applies to?
If DXC_VERSION
is the lit variable for the external validator version, I don't think the name is clear at all. It makes me think it's the version of the dxc
driver or compiler instead.
int MaybeRunExternalValidatorAndPrintValidationOutput( | ||
const DxcOpts &opts, CComPtr<IDxcOperationResult> pCompileResult); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this unused now? Should remove it if so.
// function is deferred to whatever object was chosen to be pCompiler, | ||
// which must implement the IDxcCompiler interface. External validation | ||
// will only take place if the DXC_DXIL_DLL_PATH env var is set | ||
// correctly. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this comment on this one compile call? Why not put a comment where you initialize the DxcDllExtValidationLoader
in dxc::main
instead?
if (dllResult) | ||
std::string dllLogString; | ||
llvm::raw_string_ostream dllErrorStream(dllLogString); | ||
HRESULT dllResult = dxcSupport.initialize(dllErrorStream); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Personal preference: I really dislike that initialize
takes an error stream, instead of communicating some status enum for the caller to translate into an output message if desired.
Something like:
HRESULT dllResult = dxcSupport.initialize();
if (DXC_FAILED(dllResult)) {
switch(dxcSupport.getFailureReason()) {
// print different messages for different failure modes,
// using getDxilDllPath() as necessary.
}
return dllResult;
}
if (dxcSupport.isUsingExternalValidator()) {
// print message about using getDxilDllPath().
}
private: | ||
DxcOpts &m_Opts; | ||
SpecificDllLoader &m_dxcSupport; | ||
DxcDllExtValidationLoader &m_dxcSupport; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we use DllLoader
instead? Isn't that the point of DllLoader
as a base class?
DxcDllExtValidationLoader &m_dxcSupport; | |
DllLoader &m_dxcSupport; |
class DxvContext { | ||
private: | ||
SpecificDllLoader &m_dxcSupport; | ||
DxcDllExtValidationLoader &m_dxcSupport; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same comment as for dxc.cpp:
Shouldn't we use DllLoader
instead? Isn't that the point of DllLoader
as a base class?
DxcDllExtValidationLoader &m_dxcSupport; | |
DllLoader &m_dxcSupport; |
!Opts.VerifyDiagnostics && | ||
Opts.Preprocess.empty(); | ||
const bool ProduceFullContainer = ProduceDxModule && !Opts.CodeGenHighLevel; | ||
NeedsValidation = ProduceFullContainer && !Opts.DisableValidation; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ProduceDxModule
could be turned into a const accessor method on DxcOpts
, then dxcompilerobj.cpp
could use that instead of keeping these duplicated blocks in sync across projects. The ProduceFullContainer
is simpler based on that one and could be separate, or it could also be const accessors that use the ProduceDxModule
one.
NeedsValidation
could also have an accessor, and could encode the root signature logic below, using the new ShaderModel::ParseTargetProfile
method merged from your recently merged PR. DxcOpts
already has a IsRootSignatureProfile()
for initializing the RootSignatureOnly
flag, though I don't think that flag is used here, so it could be removed instead.
I agree with @alsepkow, it would be cleaner, plus locating it in DxcOpts
makes more sense.
This PR focuses on allowing dxc to use the existing infrastructure to use a
DxcDllExtValidationLoader
object, and load an external dxil.dll with it, via the environment variable. The validate*.test was added to verify that the dll is indeed being loaded and used for external validation, which causes a failure, since the dll's validator version is older than the required minimum validator version that the metadata indicates.